Dynamic virtual background selection for video communications

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media relate to a method for providing video communications with dynamic virtual backgrounds within a communication platform. Based on the contextual information of a meeting, the system selects a virtual background to be used for the meeting. During, a video meeting, the system generates for display, on one or more client devices, a composite video depicting the imagery of the user overlaid on imagery of the selected virtual background.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/246,571, filed Apr. 30, 2021, which is hereby incorporated byreference it its entirety.

FIELD OF INVENTION

The present invention relates generally to digital communication, andmore particularly, to systems and methods providing for videocommunications with dynamic virtual backgrounds within a videocommunication platform.

BACKGROUND

During video communications while using a video conferencing applicationor system, such as on a mobile device, smart phone or a laptop, a usermay select a virtual background that is displayed as an image or videobehind a live video feed of a user. These systems may use computervision processing techniques to extract an outline (e.g., a border ofthe user) and then display the virtual background about or around theuser during video communications with other users. However, users needto manually select a virtual background to be used for a meeting orassign a default virtual background to be used for all meetings. Whileproviding some flexibility in the choice of a virtual background to beused for video communication, contextual information of a plannedmeeting with other users is not utilized by these systems. This oftenresults in situations where users may use inconsistent virtualbackgrounds. For example, when employees of one company meet withemployees of another company, some of the employees of the same companymay be using different virtual backgrounds. The current state-of the-artvideo conferencing systems lack the ability to automatically provide forconsistency and uniformity of the virtual backgrounds used among usersparticipating in a video meeting. These current state-of-the-art systemsalso lack the ability to automatically or dynamically choose a virtualbackground to be used by meeting participants.

Thus, there is a need in the field of digital communication to create anew and useful system and method for providing video communications withdynamic virtual background selection within a video communicationplatform based on rules that evaluate contextual meeting information.The source of the problem, as discovered by the inventors, is a lack ofan ability for multiple video meeting participants to dynamically selecta virtual background prior to a meeting, where the selection of thevirtual background is based on either predefined rules and/or contextualinformation about a meeting.

SUMMARY

The invention overcomes the existing problems by providing a rule-basedselection and/or contextual-based selection of virtual backgrounds foruse during a video meeting within a video communication platform. Thesystem leverages an integrated calendar connection to maintain awarenessof various calendar details. These calendar details are evaluated by thesystem to automatically assign a particular virtual background for auser based upon predefined virtual background usage rules.

The video communication system provides functionality allowing a user toset rules or conditions where the system selects a virtual background beused by the user during video communications with other users of thesystem. These user defined rules provide for the automatic selection ofvirtual background based on contextual information of a planned meeting.The rules are evaluated by the system and compared against informationof the planned meeting. Based on information of the planned meeting, thesystem may determine a virtual background to be used by the user for theplanned meeting.

The user defined rules may include key words and Boolean logicaloperators to be compared against the information of the planned meeting.The rules may be simple comparisons of key words to be found in aplanned meeting, or complex rules using various syntax and logicalconstructs to determine whether the rule criteria or conditions are met.

The system is flexible in the sources of where meeting information areretrieved, and the manner in which a virtual background may be selectedfor use. The planned meeting information may be retrieved from acalendaring application or other databases that stores electronicmeeting information. These data sources may be resident on a user deviceand/or stored in a cloud-based storage system or service.

The rules may evaluate various data of the planned meeting information,such as whether particular users or participants are attending theplanned meeting, whether particular e-mail domain names exist, whethercertain words occur in information descriptive of meeting, whether themeeting is scheduled during certain times or dates, the number ofmeeting participants, and so forth.

If the rule criteria or conditions are met, then the system selects avirtual background that has been associated with the rule. A user, forexample, may create a rule and then the system may select or assign avirtual background to be used when the system determines that the rulecriteria is met.

The user created rules provides for consistent use of virtualbackgrounds for video communications with other users. The system mayautomatically select the same virtual background for various plannedmeetings in which the user participates. For example, users may havemany repeating meetings, or meetings with similar user participants. Auser may only need to set up a rule once, and the system then may usethe rule to select the same virtual background for future or subsequentrepeating planned meetings.

Some users may use the video communication system to engage in videocommunications in different contexts, such as using the videocommunication system for video meetings in a business context orpersonal context. The system dynamically selects a virtual backgroundfor use based on particular contextual information of the plannedmeeting. The system functionality provides for the automatic selectionof the virtual background based on the meeting context.

Methods, systems, and apparatus, including computer programs encoded oncomputer storage media relate to a method for providing videocommunications with dynamic virtual backgrounds within a communicationplatform. The system provides one or more preconfigured rules for theselection of one or more virtual backgrounds to be used for plannedvideo meetings. Based on the contextual information of a plannedvideo-based meeting, the system may select a virtual background to beused for the meeting. During the video-based meeting, the systemgenerates for display, on one or more client devices, a composite videodepicting the imagery of the user overlaid on imagery of the selectedvirtual background.

One embodiment relates to a communication system configured to perform anumber of operations. First, the system provides one or morepreconfigured rules for the selection of one or more virtual backgroundsto be used for planned meetings. The system receives or determinescontextual information about a planned meeting. The system evaluatescontextual information about the planned meeting and appliespreconfigured rules to select a virtual background to be used forvideo-based communication during the planned meeting. The system enablesvideo communications for a first user with one or more multiple otherusers. The system receives video data depicting video imagery of thefirst user. The video data may include multiple video frames of thefirst user. The system determines a boundary about the first user in thevideo frames. The boundary includes an interior portion and exteriorportion. The interior portion represents the user, and the exteriorportion represents an area that may be replaced with a virtualbackground. The system generates for display on one or more clientdevices, a composite video of the interior portion depicting imagery ofthe first user overlaid on imagery of the selected virtual background.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims and the drawings. Thedetailed description and specific examples are intended for illustrationonly and are not intended to limit the scope of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention relates generally to digital communication, andmore particularly, to systems and methods providing for containment ofsensitive data within a communication or messaging platform.

The present disclosure will become better understood from the detaileddescription and the drawings, wherein:

FIG. 1A is a diagram illustrating an exemplary environment in which someembodiments may operate.

FIG. 1B is a diagram illustrating an exemplary computer system that mayexecute instructions to perform some of the methods herein.

FIG. 2 is a diagram illustrating an example of planned meetingcontextual information.

FIG. 3 is a flow chart illustrating an exemplary method 300 that may beperformed in some embodiments.

FIG. 4 is a flow chart illustrating an exemplary method 400 that may beperformed in some embodiments.

FIG. 5A illustrates an example user interface 500 according to oneembodiment of the present disclosure.

FIG. 5B illustrates an example user interface 500 according to oneembodiment of the present disclosure.

FIG. 5C illustrates an example user interface 500 according to oneembodiment of the present disclosure.

FIG. 6 is a flow chart illustrating an exemplary method 600 that may beperformed in some embodiments.

FIG. 7 illustrates an example user interface 700 according to oneembodiment of the present disclosure.

FIG. 8 is a flow chart illustrating an exemplary method 800 that may beperformed in some embodiments.

FIG. 9 illustrates an example user interface 900 according to oneembodiment of the present disclosure.

FIG. 10 is a flow chart illustrating an exemplary method 1000 that maybe performed in some embodiments.

FIG. 11 is a diagram illustrating an exemplary computer system 1100 thatmay perform processing in some embodiments.

DETAILED DESCRIPTION

In this specification, reference is made in detail to specificembodiments of the invention. Some of the embodiments or their aspectsare illustrated in the drawings.

For clarity in explanation, the invention has been described withreference to specific embodiments, however it should be understood thatthe invention is not limited to the described embodiments. On thecontrary, the invention covers alternatives, modifications, andequivalents as may be included within its scope as defined by any patentclaims. The following embodiments of the invention are set forth withoutany loss of generality to, and without imposing limitations on, theclaimed invention. In the following description, specific details areset forth in order to provide a thorough understanding of the presentinvention. The present invention may be practiced without some or all ofthese specific details. In addition, well known features may not havebeen described in detail to avoid unnecessarily obscuring the invention.

In addition, it should be understood that steps of the exemplary methodsset forth in this exemplary patent can be performed in different ordersthan the order presented in this specification. Furthermore, some stepsof the exemplary methods may be performed in parallel rather than beingperformed sequentially. Also, the steps of the exemplary methods may beperformed in a network environment in which some steps are performed bydifferent computers in the networked environment.

Some embodiments are implemented by a computer system. A computer systemmay include a processor, a memory, and a non-transitorycomputer-readable medium. The memory and non-transitory medium may storeinstructions for performing methods and steps described herein.

Many other possibilities and options can be contemplated for this usecase and others, as will be described in further detail throughout.

I. Exemplary Environments

FIG. 1A is a diagram illustrating an exemplary environment in which someembodiments may operate. In the exemplary environment 100, a firstuser's client device 150 and one or more additional users' clientdevice(s) 160 are connected to a processing engine 102 and, optionally,a video communication platform 140. The processing engine 102 isconnected to the video communication platform 140, and optionallyconnected to one or more repositories and/or databases, including a useraccount repository 130, video content repository 132, and/or anannotation repository 134. One or more of the databases may be combinedor split into multiple databases. The first user's client device 150 andadditional users' client device(s) 160 in this environment may becomputers, and the video communication platform server 140 andprocessing engine 102 may be applications or software hosted on acomputer or multiple computers which are communicatively coupled viaremote server or locally.

The exemplary environment 100 is illustrated with only one additionaluser's client device, one processing engine, and one video communicationplatform, though in practice there may be more or fewer additionalusers' client devices, processing engines, and/or video communicationplatforms. In some embodiments, one or more of the first user's clientdevice, additional users' client devices, processing engine, and/orvideo communication platform may be part of the same computer or device.

In an embodiment, the processing engine 102 may perform the method 200(FIG. 2) or other method herein and, as a result, provide multi-pointvideo presentations with live annotations within a video communicationplatform. In some embodiments, this may be accomplished viacommunication with the first user's client device, additional users'client device(s), processing engine, video communication platform,and/or other device(s) over a network between the device(s) and anapplication server or some other network server. In some embodiments,the processing engine 102 is an application, browser extension, or otherpiece of software hosted on a computer or similar device, or is itself acomputer or similar device configured to host an application, browserextension, or other piece of software to perform some of the methods andembodiments herein.

The first user's client device 150 and additional users' clientdevice(s) 160 are devices with a display configured to presentinformation to a user of the device. In some embodiments, the firstuser's client device 150 and additional users' client device(s) 160present information in the form of a user interface (UI) with UIelements or components. In some embodiments, the first user's clientdevice 150 and additional users' client device(s) 160 send and receivesignals and/or information to the processing engine 102 and/or videocommunication platform 140. The first user's client device 150 isconfigured to perform functions related to presenting and playing backvideo, audio, documents, annotations, and other materials within a videopresentation (e.g., a virtual class, lecture, webinar, or any othersuitable video presentation) on a video communication platform. Theadditional users' client device(s) 160 are configured to viewing thevideo presentation, and in some cases, presenting material and/or videoas well. In some embodiments, first user's client device 150 and/oradditional users' client device(s) 160 include an embedded or connectedcamera which is capable of generating and transmitting video content inreal time or substantially real time. For example, one or more of theclient devices may be smartphones with built-in cameras, and thesmartphone operating software or applications may provide the ability tobroadcast live streams based on the video generated by the built-incameras. In some embodiments, the first user's client device 150 andadditional users' client device(s) are computing devices capable ofhosting and executing one or more applications or other programs capableof sending and/or receiving information. In some embodiments, the firstuser's client device 150 and/or additional users' client device(s) 160may be a computer desktop or laptop, mobile phone, virtual assistant,virtual reality or augmented reality device, wearable, or any othersuitable device capable of sending and receiving information. In someembodiments, the processing engine 102 and/or video communicationplatform 140 may be hosted in whole or in part as an application or webservice executed on the first user's client device 150 and/or additionalusers' client device(s) 160. In some embodiments, one or more of thevideo communication platform 140, processing engine 102, and firstuser's client device 150 or additional users' client devices 160 may bethe same device. In some embodiments, the first user's client device 150is associated with a first user account on the video communicationplatform, and the additional users' client device(s) 160 are associatedwith additional user account(s) on the video communication platform.

In some embodiments, optional repositories can include one or more useraccount repositories 130, meetings repositories 132, virtual backgroundrepositories 134 and/or virtual background configuration repositories136. The optional repositories function to store and/or maintain,respectively, user account information associated with the videocommunication platform 140, video content received by the videocommunication platform 140 for display within video communication, andvirtual background, which may include images, video and other graphicalvisual content. The optional database(s) may also store and/or maintainany other suitable information for the processing engine 102 or videocommunication platform 140 to perform elements of the methods andsystems herein. In some embodiments, the optional database(s) can bequeried by one or more components of system 100 (e.g., by the processingengine 102), and specific stored data in the database(s) can beretrieved.

Video communication platform 140 is a platform configured to facilitatevideo presentations and/or communication between two or more parties,such as within a video conference or virtual classroom.

FIG. 1B is a diagram illustrating an exemplary computer system 150 withsoftware modules that may execute some of the functionality describedherein.

The Planned Meeting Evaluation module 152 provides system functionalityfor the interaction and retrieval planned meeting contextual informationfrom calendaring systems and/or database, and as further describedherein.

The Virtual Background Template Configuration module 154 provides systemfunctionality for the configuration, processing, storage and managementof virtual background templates, and as further described herein.

The Virtual Background Rules Configuration module 156 provides systemfunctionality for the selection, creation, configuration, processing,storage and management of virtual background selection rules, and asfurther described herein.

Virtual Background Selection module 158 provides system functionalityfor the selection, creation, configuration, processing, storage andmanagement of particular virtual backgrounds.

Virtual Real-time Meeting Monitoring module 160 provides systemfunctionality for the monitoring and evaluation of meeting that are inprogress to identify and determine real-time video communications, audiocommunications and textual communications, and as further describedherein.

Virtual Background Synchronization module 162 provides systemfunctionality for the synchronization of virtual backgrounds among usersof a meeting, and as further described herein.

The above modules and their functions will be described in furtherdetail in relation to an exemplary methods and systems below.

Virtual Background Files

A virtual background file may be a file such as a video file, an imagefiles (e.g., a jpeg, gif, etc.), or any other type of graphics ormultimedia file. In general, a virtual background file is of a file ofany type that allows for the system to present a still graphic imageand/or a video image as a virtual background in conjunction with auser's video feed. The virtual background file may be stored on a filesystem, computer system memory, either in local storage or in aserver-based storage system or database. The system retrieves thevirtual background file to be used by the system as a virtual backgroundwhile a user is engaged in video communications with one or more otherusers. The virtual background files may be cataloged or indexed by thesystem and associated with rules for the selection of particular virtualbackground files.

Planned Meeting Contextual Information

FIG. 2 is a diagram illustrating an example of planned meetingcontextual information. The system 100 may retrieve from a meetingsrepository, such as a calendar application, information that is specificto a planned meeting event. By way of illustration, FIG. 2 depictscommon information typically stored by a calendaring application. Aplanned meeting event may include a title 202, a meeting location 204, ameeting date and time 206, a list of meeting attendees 210 and otherinformation. The listing of meeting attendees usually are identified inthe format of username@domain-name. For example, a meeting attendingmight be identified with the email address JAppleseed@zoom.com. Often auser creating a meeting, via a calendaring application, will provide ameeting title 202 to provide context for the meeting.

In some embodiments, preconfigured rules (as described herein) may useconditions and/or parameters that are evaluated against contextualinformation of a planned meeting. For instance, by way of illustration,but not limitation, some contextual information that may be evaluated bythe system include: the title of the meeting, number of meetingparticipants, the time zone of one or more meeting participants, thedescription of the meeting, the particular participant user ids, emailaddress or time zone of meeting participants.

The planned meeting contextual information may be presented, via a userinterface, such as a calendaring application user interface. In someembodiments, the calendaring application user interface may provide forthe association of a virtual background file, via user interaction withthe calendaring application user interface. For example, the calendaringapplication user may display a graphical icon where a user may upload orassign a particular virtual background to be used by one or moreattendees of a meeting.

Virtual Background Selection Rules

In some embodiments, preconfigured rules for virtual backgroundselection may be user defined, automatically generated by the system,preconfigured by the system for a user and/or the rules shared amongusers. The system provides functionality allowing a user to defineconditions and/or parameters where a virtual background file isautomatically selected for use as a virtual background for prospectivemeetings. In determining which virtual background to select, the systemevaluates contextual information of a planned meeting, and then appliesthe preconfigured rule to the contextual information to determinewhether the criteria for the selection of a virtual background are met.Where the conditions and/or parameters for the selection of a virtualbackground are met, then the system selects the virtual background to beused for user video communications.

The system may automatically generate preconfigured rules for a user forthe selection of a virtual background to be used for videocommunications. The system may evaluate historical meeting informationfrom a calendaring system, or other electronic repository having pastmeeting information for the user. The system may store information abouta prior selected virtual background that was used or assigned for usefor users for various meetings. The system may determine commonalityabout contextual information of the various meeting, and automaticallygenerate a preconfigured rule for a user to select a virtual backgroundfor prospective meetings. For example, the system may determine that auser used the same virtual background where the meeting participantsincluded a specific user email domain name, or a particular user emailaddress. The system then may create a preconfigured rule toautomatically select the same virtual background that was previouslyused, where the user has a planned meeting with any other users having asimilar email domain name and/or if the particular user email address isfound in the contextual information of a planned meeting.

In another example, the system may determine that a user typically usesthe same virtual background (such as an image with the user's companylogo) where the title of a meeting (as found in the meeting contextualinformation) is called “Company All-hands”. The system may then generatea preconfigured rule for that user where the system would evaluate thecontextual information of a planned meeting for keywords of “CompanyAll-hands” (either case-sensitive or not), and then associate thepreviously used virtual background for the planned meeting where thepreconfigured rule criteria is met. In this instance, the system wouldautomatically select the image of the user's company logo, when thetitle of a planned meeting includes the keywords “Company All-hands”.

The system provides functionality where users may share theirpreconfigured rules with other users. The system may provide a userinterface where a user may select one or more preconfigured rules tosend to one or more other users. The selection user interface may allowfor the input of user names and/or email addresses to whom thepreconfigured rules would be shared. When sharing the preconfiguredrule, the system provides the rule name, conditions and parameters, andcopy of the underlying virtual background file to the other user. Thisfunctionality provides for one user to share a preconfigured rule withthe other users. The other users may now use or modify the preconfiguredrules for their own meetings.

In some embodiments, the preconfigured rules may be given a rulepriority order or weight for consideration by the system. In otherwords, the system may receive an indication that a rule has a higherpriority than another rule. For example, the preconfigured rules may begiven a rank or priority order for evaluation by the system. In oneembodiment, a user interface may display a listing of the preconfiguredrules, and the order of the rules (for example, top to bottom) indicatethe order or priority that the system will evaluate the rule. In anotherembodiment, a user interface may receive a ranking number to indicatethe order the preconfigured rule would be evaluated by the system.

In some embodiments, the conditions and/or parameters of a preconfiguredrule may include key words to match, Boolean logic operators to findmatches (such as “and”, “or”), wild card operators and differentcharacter strings. A preconfigured rule may also include negativeoperators to preclude when a virtual background is not to be used. Forexample, a user may have personal based virtual backgrounds that areused when video conferencing with family or friends, and may not want tohave the virtual background used for company or business meetings. Theuser may identify conditions and/or parameters of when not to use aparticular virtual background for certain meeting contextualinformation. In this instance, this would preclude use of the virtualbackground from being automatically selected for those do not usemeeting situations. Also, the system may be configured to preclude theuser from manually selecting the virtual background is such situations,or at least prompt the user noting the virtual background has beenprecluded for use for the particular meeting at hand, and then allowingthe user to override the preconfigured rule is the user so desires.

In some embodiments, the system may in some instances, select two ormore virtual backgrounds that may be used for a video meeting based onthe application of two or more preconfigured rules as applied tocontextual information of a planned meeting. In such a case, the systemmay present to the user a user interface depicting two or more graphicalrepresentations of the selected two or more virtual backgrounds. Thesystem may receive a selection from a user for one of the displayedgraphical representations. The system then would use the virtualbackground associated with the received selection of the graphicalrepresentation for use as a virtual background for the user during videocommunications.

II. Exemplary Methods and User Interfaces

Virtual Background Selection

FIG. 3 is a flow chart illustrating an exemplary method 300 that may beperformed in some embodiments. In one embodiment, the system 100provides for virtual background selection using preconfigured rules. Thesystem 100 provides functionality where a user may define a rule toselect a particular virtual background to be used for a planned videomeeting. The system 100 evaluates planned meeting contextual informationand determines a specific virtual background to be used for videocommunication. In some instances, two or more rules may be applicableand result in the selection of two different virtual backgrounds foruse. In the case, the system may present a user interface displaying arepresentation of the two virtual backgrounds, and receive a selectionfor the desired virtual background to be used during videocommunications by the user.

At step 310, the system 100 provides one or more preconfigured rules forthe selection of one or more virtual backgrounds to be used for plannedvideo meetings. The preconfigured rules may be stored on a local orcloud data storage device and retrieved by the system.

At step 320, the system 100 receives contextual information about aplanned meeting. For example, the system may retrieve contextualinformation from a calendaring system or other database storingcontextual information about planned meetings.

At step 330, the system 100 selects at least one virtual background foruse by a first user for the planned meeting. The system may select avirtual background based on the application of one or more preconfiguredrules as applied to the received contextual information of the plannedmeeting.

At step 340, the system 100 receives video content depicting imagery ofthe first user, the video content having multiple video frames. Thesystem provides functionality for a user to capture and display videoimagery to other users. For example, the system may receive a videostream from digital camera depicting imagery of the first user.

At step 350, the system determines a boundary about the first user inthe video frames, wherein the boundary has an interior portion andexterior portion. The system may use various computer vision techniquesto identify the border or outline of the user depicted in the video.

At step 360, the system 100 generates, for display on one or more clientdevices, a composite video depicting the interior portion depictingimagery of the first user overlaid on imagery of the selected virtualbackground. The system combines or composites video imagery of the usershown with the selected virtual background.

FIG. 4 is a flow chart illustrating an exemplary method 400 that may beperformed in some embodiments. The method 400 show an example ofprocessing multiple preconfigured rules, where the rules are evaluatingthe meeting contextual information.

The method of evaluating meeting contextual information is illustratedfor a search and match process for multiple preconfigured rules, witheach preconfigured rule searching for and matching key words. The systemmay retrieve from a data storage device multiple preconfigured todetermine a virtual background filed to be selected for a user as avirtual background during video communications with other users. In step410, the system evaluates one or more retrieved n preconfigured rules.In step 420, the system determines particular keys words for theparticular nth rule. The system determines the rule's logical operatorsand syntax, for example, determining which key words to match to plannedmeeting contextual information. For example, one or more key words maybe required to be matched to meeting contextual information for aparticular virtual background file to be selected.

At step 430, the system compares or tries to find matches of the keywords with contextual information of a planned meeting. At step 440, thesystem determines whether or not a match was found. For each of the keywords, the system determines whether all of the key words are found. Ifall of the key words are matched to contextual information of themeeting, then the system may then select a virtual background file thatwas associated with the preconfigured rule (step 450). The system woulduse the virtual background file to provide the virtual background duringvideo communications by the user. While a simple key word match may beconfigured as a rule, more complex rules, operations, parameters andconditions may be required for a preconfigured rule. If no required keyword matches are found, then next preconfigured rule is then evaluatedagainst the contextual information, and so on until no more rules exist(looping back to step 430). The system may be configured such that aparticular virtual background file would be used if none of thepreconfigured rules apply to the contextual meeting information.

FIG. 5A illustrates an example user interface according to oneembodiment of the present disclosure. The system 100 may generate anddisplay a user interface 500 where a user may select a virtualbackground to use as a default virtual background. The user interface500 may provide a selection for virtual background configuration wherethe user interface 500 provides functionality where a user may configureone or more rules for the selection of a virtual background to be usedfor a planned video meeting.

FIG. 5B illustrates an example user interface 500 according to oneembodiment of the present disclosure. In response to the selection of“Virtual Background Configuration” text, the system 100 may display theuser interface section 530. The user interface section 530 depicts alisting of key terms and a group of virtual background from which avirtual background may be selected to be used when the key words arefound in the contextual information for a planned meeting. The listingof key words 532 may be removed by the selection of the trashcan icon tothe left of a particular key word.

The following example illustrates how a user may interact with the userinterface section 530. A user may want to set or select a virtualbackground to be used for planned meeting. The user would enter keywords into the input text box 534. In the examples 532A, 532B and 532C,a user had typed in text in the user interface input text box 534. Fromthe listing of virtual backgrounds, the user had also selected aparticular virtual background to be used for the input key words.

In one example, for the key words “ABC CO” 532A, a user may haveselected the graphical representation of virtual background 542A to beused when a planned meeting's contextual information includes the keywords “ABC CO”. As will be described further below, the system 100 mayevaluate the planned meeting contextual information to identify that theplanned meeting has the phrase “ABC CO”. In such a case, the system 100will select and utilize the virtual background that is associated withthe graphical representation of virtual background 542A to be used as avirtual background for the user during video communications with otherusers.

In another example, for the key words “Weekly Review” 532B, a user mayhave selected the graphical representation of virtual background 542B tobe used when a planned meeting's contextual information includes the keywords “Weekly Review”. As will be described further below, the system100 may evaluate the planned meeting contextual information to identifythat the planned meeting has the phrase “Weekly Review” for instance inthe title of the meeting. In such a case, the system 100 will select andutilize the virtual background that is associated with the graphicalrepresentation of virtual background 542B to be used as a virtualbackground for the user during video communications with other users.

In yet another example, for the key words “@zoom.com” 532C, a user mayhave selected the graphical representation of virtual background 542C tobe used when a planned meeting's contextual information includes the keywords “@zoom.com”. As will be described further below, the system 100may evaluate the planned meeting contextual information to identify thatthe planned meeting has the phrase “@zoom.com”, for instance in thee-mail address of one of the meeting participants. In such a case, thesystem 100 will select and utilize the virtual background that isassociated with the graphical representation of virtual background 542Cto be used as a virtual background for the user during videocommunications with other users.

FIG. 5C illustrates an example user interface according to oneembodiment of the present disclosure. In this embodiment of the userinterface 500, the system 100 may present the user interface with aninput section where the user interface 500 may receive a selection forthe name of a rule, a selection or input for a user calendar associatedwith the rule (e.g., Outlook.com, Gmail.com, iCloud.com, etc.), and oneor more key words to be matched for a planned meeting's contextualinformation. In this embodiment, the user interface may upload a virtualbackground to the system local or cloud-based storage repository inassociation with an account and/or user id of the user.

Automatic Virtual Background Rule Generation

In some embodiments, the system 100 may evaluate the historical use ofvirtual backgrounds that were previously used for a meeting, andautomatically construct rules for use of virtual backgrounds for plannedmeetings. As described previously, the system may determine that aprevious virtual background was used for many different meetings. Thesystem may perform processing to determine similarities of contextualinformation of the different meetings and derive or create rules to usethe same virtual background where a planned meeting has similarcontextual information as the historical meetings.

In this embodiment, the system may establish a connection to thecalendar account of a first user, or other database storing previousmeeting information. A calendar account of a user is capable of storinginformational details about one or more meetings. The informationaldetails, for example, may include for each meeting: a title for themeeting, participant e-mail addresses of the meeting, a date for themeeting and a time for the meeting. Also, the system 100 may storeinformation for particular virtual backgrounds that were used for pastmeetings.

The system 100 may evaluate the informational details of prior meetingsof the calendar account of the first user. The system may retrieve andgenerate a dataset include the information details for those meetingswhere the same virtual background was used.

The system 100 may generate one or more rules for the selection of avirtual background to be used for prospective meetings by the firstuser. The generated one or more rules may be generated based on anevaluation of the informational details of the prior meetings. Forexample, the system may determine that similar key words exist in thecontextual information for the meetings. The system may then generate arule where the system would use the same virtual background aspreviously used, where the key words are found in the contextualinformation for a planned meeting.

The system 100 may the determine, by applying the generated one or morerules, a virtual background to use for a planned meeting.

Virtual Background Template Configuration

FIG. 6 is a flow chart illustrating an exemplary method 600 that may beperformed in some embodiments. The system 100 provides functionality todisplay a user interface (as further described with respect to FIG. 7)where a user may define a template for the selection of one or morevirtual backgrounds to be used for a planned video meeting. The systemmay then evaluate planned meeting contextual information and select aspecific virtual background template to be used for video communication.During video communications by the user, the system would use thevirtual backgrounds configured for the selected virtual backgroundtemplate.

At step 610, the system displays a template configuration userinterface. The template configuration user interface may be configuredto receive the placement of one or more virtual background boundaryareas.

At step 620, the system receives, via the template configuration userinterface, the placement of one or more virtual background boundaryareas. The user interface may receive selection about the templateconfiguration interface of where the user would like to add a virtualbackground boundary area.

At step 630, the system receives, via the template configuration userinterface, a selection for one or more virtual backgrounds to beassociated with each of the one or more placed virtual backgroundboundary areas. For each of the virtual background boundary areas, auser may select a virtual background to be displayed within a particularvirtual background boundary area.

At step 640, the system stores, on a storage device, the user configuredvirtual background template including the location of the one or morevirtual background boundary areas about the user interface, and theinformation as to the selected one or more virtual backgrounds for therespective virtual background boundary areas.

At step 650, the system, assigns use conditions/parameters to determinewhen to use the stored virtual background template for a plannedmeeting. The use conditions may be preconfigured rules as describedherein, such as assigning keywords to be matched as to the contextualinformation of a planned meeting.

At step 660, the system based on the template use conditions/parametersas applied to the contextual information of a planned meeting, selectingthe stored virtual background template to be used for the meeting.

At step 670, the system 100 displays, via a user interface, to one ormore meeting participants during a video meeting, the one or morevirtual backgrounds assigned to the virtual background template.

Virtual Background Template

FIG. 7 illustrates an example user interface 700 according to oneembodiment of the present disclosure. The system 100 may generate anddisplay a user interface 700 where a user may configure a virtualbackground template to use for video communications with other users.The user interface 700 may provide functionality for the selection andplacement of sections and/or borders (i.e., virtual background boundaryareas) on the user interface 700 for the display of one or more virtualbackgrounds. The virtual background template during configuration mayoptionally display a video feed of a user 708 during the configurationof the virtual background template.

The system may generate and display the user interface 700 which allowsa user to design or configure one or more virtual backgrounds to bedisplayed during a video communication with other users. A user mayselect, draw, move, delete and/or resize one or more virtual backgroundboundary areas on the user interface 700 (e.g., using a selection toolvia a mouse input, touch input or other input device). The userinterface 700 may receive the placement of one or more virtualbackground boundary area onto the user interface. For example, the userinterface 700 depicts three virtual background boundary areas showing adesired placement of three different virtual background boundary areas(represented by the dashed borders 702, 704, 706). A virtual backgroundboundary area may be layered such that top layer (or most forwardpositioned) virtual background boundary area displays a virtualbackground over another virtual background during the videocommunications.

A virtual background boundary area may be associated with a particularvirtual background to be displayed during video communications when thevirtual background template is selected for use for a meeting. As anexample, the virtual background boundary area 704 is associated with avirtual background of a graphical image of the word “ZOOM”.

The associated virtual backgrounds for the respective virtual backgroundboundary areas may be stored collectively together as a virtualbackground template. The virtual background template may include thelocations, shape and size of the virtual background boundary areas, andan indication such as a file name, link, identifier or other referencefor a specific virtual background to be used in association with thevirtual background boundary area. The virtual background template may bestored on a local data storage device and/or a remote storage service.In some embodiments, the virtual background files may be stored alongwith the virtual background as a collective container of the data,information and files needed to use or recreate the virtual backgroundtemplate.

A particular virtual background template may be associated withpre-configuration rules (i.e., similar to the virtual background ruleconfiguration as discussed previously). Additionally, each of theindividual virtual background boundary areas may be associated with thesame or different pre-configuration rules. A virtual background boundaryarea may be associated with a rule that if key words are matched in thecontextual information of a planned meeting, then the virtual backgroundboundary area would be displayed. For example, if a preconfigured ruleincludes the keywords “zoom.com”, and the contextual informationincludes a reference to zoom.com, such as a user email address, then thevirtual background boundary area 704 would be initiated and becomeactive during the video meeting, and the system would display theassociated virtual background (e.g., the graphical image of the word“ZOOM”.)

In some embodiments, each of the virtual background boundary areas maybe associated with different or similar preconfigured rules. Also, aparticular virtual background boundary area may be associated withmultiple preconfigured rules. For example, the virtual backgroundboundary area 704 may also be associated with the keywords “Baseball”,and virtual background (such as a logo of one's favorite baseball team)may be associated with the keywords and the virtual background boundaryarea 704. In this instance, if a planned meeting includes contextualinformation include a keyword for baseball, then the system 100 wouldpresent associated virtual background for the associated baseball teamlogo. This functionality allows a user to configure a dynamic virtualbackground template that displays different virtual backgrounds for thesame virtual background boundary areas based on a preconfigured rule andthe informational content of a planned meeting.

In some embodiments, the user interface 700 may receive a textual inputdirectly into the virtual background boundary area. For example, theuser interface 700 may receive the input for the word Hobby,CompanyName, etc. The system 100 may receive and store user input tagsfor different virtual backgrounds. For example, system 100 may receive atag from a user input, associating a particular virtual background withthe tag. In one instance an image to be used as a virtual background maybe tagged with the word “Hobby” and another image tagged with a companyname. Then for a planned video meeting, the virtual background would beused where the word “Hobby” or the company name is found in thecontextual information for the meeting.

Additionally, the virtual background boundary area may be set as random.In this case, a different virtual background may be automatically chosesby the system 100 for each new video meeting in which the userparticipates.

Virtual Background Synchronization

FIG. 8 is a flow chart illustrating an exemplary method 800 that may beperformed in some embodiments. The system 100 provides functionality todisplay a user interface where a first user may select a virtualbackground to be used for a video meeting. The system 100 willsynchronize the virtual background of the other user with the virtualbackground selected by the first user. The system 100 may optionallytransmit a file for the virtual background to those users that do notalready have access to the same virtual background selected by the firstuser. During a video meeting the selected virtual background would beused by the first user and at least one other user.

At step 810, the system receives a selection of a first user for the useof a virtual background to be used for a video meeting. For example, auser may select which virtual background should be used for a meeting.Alternatively, the system may use a preconfigured rule to select avirtual background to be used for the meeting.

At step 820, the system synchronizes the selected virtual background tobe used by one or more other users of the video meeting. The systemprovides for the selection of one or more other user to use the selectedvirtual background. This functionality provides, for example, a meetingadmin, meeting host, etc., to enforce a common virtual background to beused by the other meeting participants.

At step 830, the system optionally transmits a file of the selectedvirtual background to the one or more other users if the other one ormore users do not have access to the virtual background selected by thefirst users. For example, a user may not have access to virtualbackground selected by the first user. In this instance, the system maytransmit a file of the selected virtual background to the other user.The other user may the receive and store the received virtual backgroundon their local or cloud data storage device.

At step 840, during a video meeting, the system displays for the firstuser, and each of the one or more other users of the video meeting, theselected virtual background. The first user and the other users woulddisplay during the video meeting the virtual background as selected bythe first user.

FIG. 9 illustrates an example user interface according to one embodimentof the present disclosure. In some embodiments, the user interfaceallows the presentation of different virtual backgrounds for selectionby a user. The system retrieves from a data storage device one or morevirtual background files that correspond to preconfigured rules for theuser. The displayed listing of virtual backgrounds for one user may bedifferent than the listing of virtual backgrounds for other users. Theselection and display of the virtual backgrounds may depend upon thepreconfigured rules for each user and on the contextual information ofthe meeting in which the users participate.

In some embodiments, the system synchronizes the displayed virtualbackgrounds and displays at least a group of similar virtual backgroundsfor each user participating in the meeting. For example, the system maydetermine that two or more users have a similar virtual background thatmay be chosen based on one or more preconfigured rules for the selectionof a virtual background based on the contextual information of themeeting. In such instances, where the system determines that each of theusers have a same virtual background that may be used for the videomeeting, the system may automatically select for use for those users thesame virtual background to be used during the meeting.

In some embodiments, a user may select a virtual background to be usedby other meeting participants. The system provides functionalityallowing a user, such as a host or co-host of a meeting, to select oneof the virtual backgrounds to be used by all of the meetingparticipants, a group of meeting participants, and/or or specificidentified meeting participants. For example, this functionality isespecially useful in that the system allows a host to select whichvirtual background would be used for the meeting for some or all of theusers. In a situation where employees of the same company are meetingwith another company, a user of the company hosting the meeting canensure a consistent virtual background to be used by all of the users ofthe company participating on a video conference call. Where a userselects a virtual background for use by certain users or all of users,the system would cause the selection of the particular virtualbackgrounds to be used by each of the certain users. When the certainusers participate in a video meeting, each of their virtual backgroundswould be uniformly displayed with the same virtual background.

In some embodiments, a user, such as a meeting host or admin, may assigna virtual background to be used by the particular user. As describedwith respect the planned meeting contextual information, a primary usermay assign a virtual background to be used directly for a user. Duringthe video meeting, the primary user assigned virtual background wouldthen be used by the respective users. In alternative embodiments, theprimary user may assign a particular virtual background to be used byall meeting participants. In other words, a virtual background may beassigned at a meeting level where each meeting participant would bedisplayed with the same virtual background.

Dynamic Virtual Background Selection

FIG. 10 is a flow chart illustrating an exemplary method 1000 that maybe performed in some embodiments. In some embodiments, the system mayevaluate meeting content and select a virtual background for a userbased on the evaluated meeting context.

Speech-to-Text and Selection of Virtual Backgrounds

In one embodiment, the system may convert speech-to-text and based on anevaluation of the text determine a virtual background to use, ordynamically change to during the meeting.

In reference to FIG. 10, at step 1010, the system 100 assigns a virtualbackground to be used for a video meeting, the assignment based on basedon key words or character strings of textual content of the meeting.

At step 1020, the system 100 initiates a video meeting, the videomeeting including two or more video meeting participants, wherein afirst meeting participant utilizes a first virtual background fordisplay.

At step 1030, the system 100 convert audio speech of the two or morevideo meeting participants to textual data.

At step 1040, the system 100 evaluates the textual data for key words orcharacter strings.

At step 1050, the system 100 selects the virtual background for useduring the video meeting where the key words or character strings matchtextual data.

At step 1060, the system 100 displays for the first user, the selectedvirtual background during the video meeting or for at least a portion ofthe video meeting.

Object Detection of User Video and Selection of Virtual Backgrounds

In other embodiments, the system 100 may evaluate the video stream of auser and determine a virtual background to use for video communicationsbased on text found in the video stream of a user. For example, thesystem 100 may receive video frames, evaluate the video frames andidentify one or more objects in the video frames. The system 100 maydetermine whether an object includes textual information, such as thename on a hat and/or a logo on a t-shirt. The system 100 may thensuggest to a user or automatically select a virtual background to usefor video communications that is associated with the text identified inone or more objects of the video frames.

In other embodiments, the system 100 may evaluate the video stream of auser and determine a virtual background to use for video communicationsbased on motions of the user. For example, the system 100 may receivevideo frames, evaluate the video frames and identify user motions in thevideo frames. The system 100 may determine whether a particular motionhas been made (for example, hand signing or performing a gesture withthe hands, or head movement). The motion may be associated with aparticular virtual background to be used when the motion is identifiedby the system. For example, the system 100 may detect that a user isperforming the motion of waving goodbye, the system 100 may in responsedisplay a virtual background associated with the detected waving goodbyemotion.

Exemplary Computer System

FIG. 11 is a diagram illustrating an exemplary computer that may performprocessing in some embodiments. Exemplary computer 1100 may performoperations consistent with some embodiments. The architecture ofcomputer 1100 is exemplary. Computers can be implemented in a variety ofother ways. A wide variety of computers can be used in accordance withthe embodiments herein.

Processor 1101 may perform computing functions such as running computerprograms. The volatile memory 1102 may provide temporary storage of datafor the processor 1101. RAM is one kind of volatile memory. Volatilememory typically requires power to maintain its stored information.Storage 1103 provides computer storage for data, instructions, and/orarbitrary information. Non-volatile memory, which can preserve data evenwhen not powered and including disks and flash memory, is an example ofstorage. Storage 1103 may be organized as a file system, database, or inother ways. Data, instructions, and information may be loaded fromstorage 1103 into volatile memory 1102 for processing by the processor1101.

The computer 1100 may include peripherals 1105. Peripherals 1105 mayinclude input peripherals such as a keyboard, mouse, trackball, videocamera, microphone, and other input devices. Peripherals 1105 may alsoinclude output devices such as a display. Peripherals 1105 may includeremovable media devices such as CD-R and DVD-R recorders/players.Communications device 1106 may connect the computer 100 to an externalmedium. For example, communications device 1106 may take the form of anetwork adapter that provides communications to a network. A computer1100 may also include a variety of other devices 1104. The variouscomponents of the computer 1100 may be connected by a connection mediumsuch as a bus, crossbar, or network.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “identifying” or “determining” or “executing” or“performing” or “collecting” or “creating” or “sending” or the like,refer to the action and processes of a computer system, or similarelectronic computing device, that manipulates and transforms datarepresented as physical (electronic) quantities within the computersystem's registers and memories into other data similarly represented asphysical quantities within the computer system memories or registers orother such information storage devices.

The present disclosure also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for theintended purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but not limited to, any type of diskincluding floppy disks, optical disks, CD-ROMs, and magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, each coupled to a computer system bus.

Various general purpose systems may be used with programs in accordancewith the teachings herein, or it may prove convenient to construct amore specialized apparatus to perform the method. The structure for avariety of these systems will appear as set forth in the descriptionabove. In addition, the present disclosure is not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the disclosure as described herein.

The present disclosure may be provided as a computer program product, orsoftware, that may include a machine-readable medium having storedthereon instructions, which may be used to program a computer system (orother electronic devices) to perform a process according to the presentdisclosure. A machine-readable medium includes any mechanism for storinginformation in a form readable by a machine (e.g., a computer). Forexample, a machine-readable (e.g., computer-readable) medium includes amachine (e.g., a computer) readable storage medium such as a read onlymemory (“ROM”), random access memory (“RAM”), magnetic disk storagemedia, optical storage media, flash memory devices, etc.

In the foregoing disclosure, implementations of the disclosure have beendescribed with reference to specific example implementations thereof. Itwill be evident that various modifications may be made thereto withoutdeparting from the broader spirit and scope of implementations of thedisclosure as set forth in the following claims. The disclosure anddrawings are, accordingly, to be regarded in an illustrative senserather than a restrictive sense.

What is claimed is:
 1. A computer-implemented method for providingdynamic virtual backgrounds within a video communication system, themethod comprising: receiving contextual information about a meeting;applying at least one virtual background selection rule to the receivedcontextual information to determine a virtual background for use by afirst client device during the meeting; receiving video contentdepicting imagery of a first user of the first client device, the videocontent having multiple video frames; and generating, for display on atleast one client device, a composite video depicting imagery of thefirst user overlaid on imagery of the virtual background.
 2. The methodof claim 1, wherein the contextual information comprises at least atitle of the meeting, a number of meeting participants, a time zone ofone or more meeting participants or a description of the meeting.
 3. Thecomputer-implemented method of claim 1, wherein the contextualinformation comprises information of a planned meeting obtained from ameetings repository.
 4. The computer-implemented method of claim 1,wherein the virtual background selection rule is one of a user definedselection rule, a selection rule automatically generated by thecommunication system, or a selection rule preconfigured by thecommunication system.
 5. The computer-implemented method of claim 1,wherein the virtual background selection rule is automatically generatedby the communication system based on a prior selected virtual backgroundfor a previous meeting having similar contextual information.
 6. Thecomputer-implemented method of claim 1, further comprising: applying thevirtual background rule to select the virtual background based on apriority ranking of the virtual background rule.
 7. Thecomputer-implemented method of claim 1, wherein the virtual backgroundrule is one of multiple virtual background rules applied to thecontextual information to select the virtual background for the meeting.8. Non-transitory computer storage that stores executable programinstructions that, when executed by one or more computing devices,configure the one or more computing devices to perform operationscomprising: receiving contextual information about a meeting; applyingat least one virtual background selection rule to the receivedcontextual information to determine a virtual background for use by afirst client device during the meeting; receiving video contentdepicting imagery of a first user of the first client device, the videocontent having multiple video frames; and generating, for display on atleast one client device, a composite video depicting imagery of thefirst user overlaid on imagery of the virtual background.
 9. Thenon-transitory computer storage of claim 8, wherein the contextualinformation comprises at least a title of the meeting, a number ofmeeting participants, a time zone of one or more meeting participants ora description of the meeting.
 10. The non-transitory computer storage ofclaim 8, wherein the contextual information comprises information of aplanned meeting obtained from a meetings repository.
 11. Thenon-transitory computer storage of claim 8, wherein the virtualbackground selection rule is one of a user defined selection rule, aselection rule automatically generated by the communication system, or aselection rule preconfigured by the communication system.
 12. Thenon-transitory computer storage of claim 8, wherein the virtualbackground selection rule is automatically generated by thecommunication system based on a prior selected virtual background for aprevious meeting having similar contextual information.
 13. Thenon-transitory computer storage of claim 8, further comprising theoperations of: applying the virtual background rule to select thevirtual background based on a priority ranking of the virtual backgroundrule.
 14. The non-transitory computer storage of claim 8, wherein thevirtual background rule is one of multiple virtual background rulesapplied to the contextual information to select the virtual backgroundfor the meeting.
 15. A system comprising one or more processorsconfigured to perform the operations of: receiving contextualinformation about a meeting; applying at least one virtual backgroundselection rule to the received contextual information to determine avirtual background for use by a first client device during the meeting;receiving video content depicting imagery of a first user of the firstclient device, the video content having multiple video frames; andgenerating, for display on at least one client device, a composite videodepicting imagery of the first user overlaid on imagery of the virtualbackground.
 16. The system of claim 15, wherein the contextualinformation comprises at least a title of the meeting, a number ofmeeting participants, a time zone of one or more meeting participants ora description of the meeting.
 17. The system of claim 15, wherein thecontextual information comprises information of a planned meetingobtained from a meetings repository.
 18. The system of claim 15, whereinthe virtual background selection rule is one of a user defined selectionrule, a selection rule automatically generated by the communicationsystem, or a selection rule preconfigured by the communication system.19. The system of claim 15, wherein the virtual background selectionrule is automatically generated by the communication system based on aprior selected virtual background for a previous meeting having similarcontextual information.
 20. The system of claim 15, further comprisingthe operations of: applying the virtual background rule to select thevirtual background based on a priority ranking of the virtual backgroundrule.